{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# JSON\n",
    "\n",
    "Let's look at how to load files with the ```.json``` extension using a loader.\n",
    "\n",
    "- Author: [leebeanbin](https://github.com/leebeanbin)\n",
    "- Peer Review : [syshin0116](https://github.com/syshin0116), [Teddy Lee](https://github.com/teddylee777)\n",
    "- Proofread : [JaeJun Shim](https://github.com/kkam-dragon)\n",
    "- This is a part of [LangChain Open Tutorial](https://github.com/LangChain-OpenTutorial/LangChain-OpenTutorial/tree/main/06-DocumentLoader)\n",
    "\n",
    "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/LangChain-OpenTutorial/LangChain-OpenTutorial/blob/main/06-DocumentLoader/09-JSONLoader.ipynb)[![Open in GitHub](https://img.shields.io/badge/Open%20in%20GitHub-181717?style=flat-square&logo=github&logoColor=white)](https://github.com/LangChain-OpenTutorial/LangChain-OpenTutorial/blob/main/06-DocumentLoader/09-JSONLoader.ipynb)\n",
    "\n",
    "## Overview\n",
    "This tutorial demonstrates how to use LangChain's ```JSONLoader``` to load and process JSON files. We'll explore how to extract specific data from structured JSON files using jq-style queries.\n",
    "\n",
    "### Table of Contents\n",
    "- [Environment Set up](#environment-setup)\n",
    "- [JSON](#json)\n",
    "- [Overview](#overview)\n",
    "- [Generate JSON Data](#generate-json-data)\n",
    "- [JSONLoader](#jsonloader)\n",
    "  \n",
    "When you want to extract values under the content field within the message key of JSON data, you can easily do this using ```JSONLoader``` as shown below.\n",
    "\n",
    "\n",
    "### References\n",
    "- https://python.langchain.com/docs/how_to/document_loader_json/\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Environment Setup\n",
    "\n",
    "Set up the environment. You may refer to [Environment Setup](https://wikidocs.net/257836) for more details.\n",
    "\n",
    "[Note]\n",
    "- ```langchain-opentutorial``` is a package that provides a set of easy-to-use environment setup, useful functions and utilities for tutorials. \n",
    "- You can check out the [```langchain-opentutorial```](https://github.com/LangChain-OpenTutorial/langchain-opentutorial-pypi) for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-17T13:43:22.351596Z",
     "start_time": "2025-01-17T13:43:21.461723Z"
    }
   },
   "outputs": [],
   "source": [
    "%%capture --no-stderr\n",
    "%pip install langchain-opentutorial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-17T13:43:25.396451Z",
     "start_time": "2025-01-17T13:43:24.717342Z"
    }
   },
   "outputs": [],
   "source": [
    "# Install required packages\n",
    "from langchain_opentutorial import package\n",
    "\n",
    "package.install(\n",
    "    [\n",
    "        \"langsmith\",\n",
    "        \"langchain\",\n",
    "        \"langchain_community\",\n",
    "        \"langchain_openai\"\n",
    "    ],\n",
    "    verbose=False,\n",
    "    upgrade=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-17T13:43:29.100059Z",
     "start_time": "2025-01-17T13:43:28.323428Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: rq in /Users/leejungbin/Library/Caches/pypoetry/virtualenvs/langchain-opentutorial-LGorndcz-py3.11/lib/python3.11/site-packages (2.1.0)\r\n",
      "Requirement already satisfied: click>=5 in /Users/leejungbin/Library/Caches/pypoetry/virtualenvs/langchain-opentutorial-LGorndcz-py3.11/lib/python3.11/site-packages (from rq) (8.1.8)\r\n",
      "Requirement already satisfied: redis>=3.5 in /Users/leejungbin/Library/Caches/pypoetry/virtualenvs/langchain-opentutorial-LGorndcz-py3.11/lib/python3.11/site-packages (from rq) (5.2.1)\r\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install rq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set environment variables\n",
    "from langchain_opentutorial import set_env\n",
    "\n",
    "set_env(\n",
    "    {\n",
    "        \"OPENAI_API_KEY\": \"\",\n",
    "        \"LANGCHAIN_API_KEY\": \"\",\n",
    "        \"LANGCHAIN_TRACING_V2\": \"true\",\n",
    "        \"LANGCHAIN_ENDPOINT\": \"https://api.smith.langchain.com\",\n",
    "        \"LANGCHAIN_PROJECT\": \"09-JSONLoader\",\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can alternatively set ```OPENAI_API_KEY``` in ```.env``` file and load it. \n",
    "\n",
    "[Note] This is not necessary if you've already set ```OPENAI_API_KEY``` in previous steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load environment variables\n",
    "# Reload any variables that need to be overwritten from the previous cell\n",
    "\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "load_dotenv(override=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate JSON Data\n",
    "\n",
    "---\n",
    "\n",
    "If you want to generate JSON data, you can use the following code.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-17T13:44:53.491736Z",
     "start_time": "2025-01-17T13:44:16.568770Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated and saved JSON data:\n",
      "{'people': [{'address': {'city': 'Springfield',\n",
      "                         'country': 'USA',\n",
      "                         'state': 'IL',\n",
      "                         'street': '123 Maple St',\n",
      "                         'zip': '62701'},\n",
      "             'age': 28,\n",
      "             'contact_details': {'email': 'alice.johnson@example.com',\n",
      "                                 'phone': '+1-555-123-4567'},\n",
      "             'interesting_information': {'pet': {'breed': 'Golden Retriever',\n",
      "                                                 'name': 'Buddy',\n",
      "                                                 'type': 'dog'},\n",
      "                                         'travel_history': [{'country': 'Japan',\n",
      "                                                             'year': 2019},\n",
      "                                                            {'country': 'Italy',\n",
      "                                                             'year': 2021}]},\n",
      "             'name': 'Alice Johnson',\n",
      "             'personal_preferences': {'favorite_food': 'sushi',\n",
      "                                      'hobbies': ['reading',\n",
      "                                                  'hiking',\n",
      "                                                  'photography'],\n",
      "                                      'music_genres': ['jazz',\n",
      "                                                       'classical',\n",
      "                                                       'indie']}},\n",
      "            {'address': {'city': 'Denver',\n",
      "                         'country': 'USA',\n",
      "                         'state': 'CO',\n",
      "                         'street': '456 Oak Ave',\n",
      "                         'zip': '80202'},\n",
      "             'age': 34,\n",
      "             'contact_details': {'email': 'michael.smith@example.com',\n",
      "                                 'phone': '+1-555-234-5678'},\n",
      "             'interesting_information': {'pet': {'breed': 'Siamese',\n",
      "                                                 'name': 'Whiskers',\n",
      "                                                 'type': 'cat'},\n",
      "                                         'volunteering': {'organization': 'Local '\n",
      "                                                                          'Food '\n",
      "                                                                          'Bank',\n",
      "                                                          'years_active': 5}},\n",
      "             'name': 'Michael Smith',\n",
      "             'personal_preferences': {'favorite_food': 'pizza',\n",
      "                                      'hobbies': ['cycling',\n",
      "                                                  'cooking',\n",
      "                                                  'gaming'],\n",
      "                                      'music_genres': ['rock',\n",
      "                                                       'pop',\n",
      "                                                       'hip-hop']}},\n",
      "            {'address': {'city': 'Austin',\n",
      "                         'country': 'USA',\n",
      "                         'state': 'TX',\n",
      "                         'street': '789 Pine Rd',\n",
      "                         'zip': '73301'},\n",
      "             'age': 22,\n",
      "             'contact_details': {'email': 'emily.davis@example.com',\n",
      "                                 'phone': '+1-555-345-6789'},\n",
      "             'interesting_information': {'pet': None,\n",
      "                                         'study': {'graduation_year': 2024,\n",
      "                                                   'major': 'Fine Arts',\n",
      "                                                   'university': 'University '\n",
      "                                                                 'of Texas'}},\n",
      "             'name': 'Emily Davis',\n",
      "             'personal_preferences': {'favorite_food': 'tacos',\n",
      "                                      'hobbies': ['painting',\n",
      "                                                  'traveling',\n",
      "                                                  'yoga'],\n",
      "                                      'music_genres': ['country',\n",
      "                                                       'folk',\n",
      "                                                       'dance']}},\n",
      "            {'address': {'city': 'Seattle',\n",
      "                         'country': 'USA',\n",
      "                         'state': 'WA',\n",
      "                         'street': '101 Birch Blvd',\n",
      "                         'zip': '98101'},\n",
      "             'age': 45,\n",
      "             'contact_details': {'email': 'david.brown@example.com',\n",
      "                                 'phone': '+1-555-456-7890'},\n",
      "             'interesting_information': {'career': {'job_title': 'Software '\n",
      "                                                                 'Engineer',\n",
      "                                                    'years_experience': 20},\n",
      "                                         'pet': {'breed': 'Canary',\n",
      "                                                 'name': 'Tweety',\n",
      "                                                 'type': 'bird'}},\n",
      "             'name': 'David Brown',\n",
      "             'personal_preferences': {'favorite_food': 'steak',\n",
      "                                      'hobbies': ['golf', 'reading', 'fishing'],\n",
      "                                      'music_genres': ['blues',\n",
      "                                                       'classic rock',\n",
      "                                                       'jazz']}},\n",
      "            {'address': {'city': 'Miami',\n",
      "                         'country': 'USA',\n",
      "                         'state': 'FL',\n",
      "                         'street': '202 Cedar Ct',\n",
      "                         'zip': '33101'},\n",
      "             'age': 39,\n",
      "             'contact_details': {'email': 'sophia.wilson@example.com',\n",
      "                                 'phone': '+1-555-567-8901'},\n",
      "             'interesting_information': {'pet': {'breed': 'Bulldog',\n",
      "                                                 'name': 'Max',\n",
      "                                                 'type': 'dog'},\n",
      "                                         'travel_history': [{'country': 'Spain',\n",
      "                                                             'year': 2018},\n",
      "                                                            {'country': 'Brazil',\n",
      "                                                             'year': 2020}]},\n",
      "             'name': 'Sophia Wilson',\n",
      "             'personal_preferences': {'favorite_food': 'paella',\n",
      "                                      'hobbies': ['dancing',\n",
      "                                                  'gardening',\n",
      "                                                  'cooking'],\n",
      "                                      'music_genres': ['latin',\n",
      "                                                       'pop',\n",
      "                                                       'salsa']}}]}\n"
     ]
    }
   ],
   "source": [
    "from langchain import PromptTemplate\n",
    "from langchain_openai import ChatOpenAI\n",
    "from pathlib import Path\n",
    "from dotenv import load_dotenv\n",
    "from pprint import pprint\n",
    "import json\n",
    "import os\n",
    "\n",
    "# Load .env file\n",
    "load_dotenv()\n",
    "\n",
    "# Initialize ChatOpenAI\n",
    "llm = ChatOpenAI(\n",
    "    model=\"gpt-4o-mini\",\n",
    "    temperature=0.7,\n",
    "    model_kwargs={\"response_format\": {\"type\": \"json_object\"}}\n",
    ")\n",
    "\n",
    "# Create prompt template\n",
    "prompt = PromptTemplate(\n",
    "    input_variables=[],\n",
    "    template=\"\"\"Generate a JSON array containing detailed personal information for 5 people. \n",
    "        Include various fields like name, age, contact details, address, personal preferences, and any other interesting information you think would be relevant.\"\"\"\n",
    ")\n",
    "\n",
    "# Create and invoke runnable sequence using the new pipe syntax\n",
    "response = (prompt | llm).invoke({})\n",
    "generated_data = json.loads(response.content)\n",
    "\n",
    "# Save to JSON file\n",
    "current_dir = Path().absolute()\n",
    "data_dir = current_dir / \"data\"\n",
    "data_dir.mkdir(exist_ok=True)\n",
    "\n",
    "file_path = data_dir / \"people.json\"\n",
    "with open(file_path, \"w\", encoding=\"utf-8\") as f:\n",
    "    json.dump(generated_data, f, ensure_ascii=False, indent=2)\n",
    "\n",
    "print(\"Generated and saved JSON data:\")\n",
    "pprint(generated_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The case of loading JSON data is as follows when you want to load your own JSON data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'people': [{'address': {'city': 'Springfield',\n",
      "                         'country': 'USA',\n",
      "                         'state': 'IL',\n",
      "                         'street': '123 Maple St',\n",
      "                         'zip': '62704'},\n",
      "             'age': 28,\n",
      "             'contact': {'email': 'alice.johnson@example.com',\n",
      "                         'phone': '+1-555-0123',\n",
      "                         'social_media': {'linkedin': 'linkedin.com/in/alicejohnson',\n",
      "                                          'twitter': '@alice_j'}},\n",
      "             'interesting_fact': 'Alice has traveled to over 15 countries and '\n",
      "                                 'speaks 3 languages.',\n",
      "             'name': {'first': 'Alice', 'last': 'Johnson'},\n",
      "             'personal_preferences': {'favorite_food': 'Italian',\n",
      "                                      'hobbies': ['Reading',\n",
      "                                                  'Hiking',\n",
      "                                                  'Cooking'],\n",
      "                                      'music_genre': 'Jazz',\n",
      "                                      'travel_destinations': ['Japan',\n",
      "                                                              'Italy',\n",
      "                                                              'Canada']}},\n",
      "            {'address': {'city': 'Metropolis',\n",
      "                         'country': 'USA',\n",
      "                         'state': 'NY',\n",
      "                         'street': '456 Oak Ave',\n",
      "                         'zip': '10001'},\n",
      "             'age': 34,\n",
      "             'contact': {'email': 'bob.smith@example.com',\n",
      "                         'phone': '+1-555-0456',\n",
      "                         'social_media': {'linkedin': 'linkedin.com/in/bobsmith',\n",
      "                                          'twitter': '@bobsmith34'}},\n",
      "             'interesting_fact': 'Bob is an avid gamer and has competed in '\n",
      "                                 'several national tournaments.',\n",
      "             'name': {'first': 'Bob', 'last': 'Smith'},\n",
      "             'personal_preferences': {'favorite_food': 'Mexican',\n",
      "                                      'hobbies': ['Photography',\n",
      "                                                  'Cycling',\n",
      "                                                  'Video Games'],\n",
      "                                      'music_genre': 'Rock',\n",
      "                                      'travel_destinations': ['Brazil',\n",
      "                                                              'Australia',\n",
      "                                                              'Germany']}},\n",
      "            {'address': {'city': 'Gotham',\n",
      "                         'country': 'USA',\n",
      "                         'state': 'NJ',\n",
      "                         'street': '789 Pine Rd',\n",
      "                         'zip': '07001'},\n",
      "             'age': 45,\n",
      "             'contact': {'email': 'charlie.davis@example.com',\n",
      "                         'phone': '+1-555-0789',\n",
      "                         'social_media': {'linkedin': 'linkedin.com/in/charliedavis',\n",
      "                                          'twitter': '@charliedavis45'}},\n",
      "             'interesting_fact': 'Charlie has a small farm where he raises '\n",
      "                                 'chickens and grows organic vegetables.',\n",
      "             'name': {'first': 'Charlie', 'last': 'Davis'},\n",
      "             'personal_preferences': {'favorite_food': 'Barbecue',\n",
      "                                      'hobbies': ['Gardening',\n",
      "                                                  'Fishing',\n",
      "                                                  'Woodworking'],\n",
      "                                      'music_genre': 'Country',\n",
      "                                      'travel_destinations': ['Canada',\n",
      "                                                              'New Zealand',\n",
      "                                                              'Norway']}},\n",
      "            {'address': {'city': 'Star City',\n",
      "                         'country': 'USA',\n",
      "                         'state': 'CA',\n",
      "                         'street': '234 Birch Blvd',\n",
      "                         'zip': '90001'},\n",
      "             'age': 22,\n",
      "             'contact': {'email': 'dana.lee@example.com',\n",
      "                         'phone': '+1-555-0111',\n",
      "                         'social_media': {'linkedin': 'linkedin.com/in/danalee',\n",
      "                                          'twitter': '@danalee22'}},\n",
      "             'interesting_fact': 'Dana is a dance instructor and has won '\n",
      "                                 'several local competitions.',\n",
      "             'name': {'first': 'Dana', 'last': 'Lee'},\n",
      "             'personal_preferences': {'favorite_food': 'Thai',\n",
      "                                      'hobbies': ['Dancing',\n",
      "                                                  'Sketching',\n",
      "                                                  'Traveling'],\n",
      "                                      'music_genre': 'Pop',\n",
      "                                      'travel_destinations': ['Thailand',\n",
      "                                                              'France',\n",
      "                                                              'Spain']}},\n",
      "            {'address': {'city': 'Central City',\n",
      "                         'country': 'USA',\n",
      "                         'state': 'TX',\n",
      "                         'street': '345 Cedar St',\n",
      "                         'zip': '75001'},\n",
      "             'age': 31,\n",
      "             'contact': {'email': 'ethan.garcia@example.com',\n",
      "                         'phone': '+1-555-0999',\n",
      "                         'social_media': {'linkedin': 'linkedin.com/in/ethangarcia',\n",
      "                                          'twitter': '@ethangarcia31'}},\n",
      "             'interesting_fact': 'Ethan runs a popular travel blog where he '\n",
      "                                 'shares his adventures and culinary '\n",
      "                                 'experiences.',\n",
      "             'name': {'first': 'Ethan', 'last': 'Garcia'},\n",
      "             'personal_preferences': {'favorite_food': 'Indian',\n",
      "                                      'hobbies': ['Running',\n",
      "                                                  'Travel Blogging',\n",
      "                                                  'Cooking'],\n",
      "                                      'music_genre': 'Hip-Hop',\n",
      "                                      'travel_destinations': ['India',\n",
      "                                                              'Italy',\n",
      "                                                              'Mexico']}}]}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "from pathlib import Path\n",
    "from pprint import pprint\n",
    "\n",
    "\n",
    "file_path = \"data/people.json\"\n",
    "data = json.loads(Path(file_path).read_text())\n",
    "\n",
    "pprint(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'dict'>\n"
     ]
    }
   ],
   "source": [
    "print(type(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ```JSONLoader```\n",
    "\n",
    "---\n",
    "\n",
    "When you want to extract values under the content field within the message key of JSON data, you can easily do this using ```JSONLoader``` as shown below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic Usage\n",
    "\n",
    "This usage shows off how to execute load JSON and print what I get from"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-17T13:45:23.320964Z",
     "start_time": "2025-01-17T13:45:23.227757Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 1}, page_content='{\"name\": \"Alice Johnson\", \"age\": 28, \"contact_details\": {\"email\": \"alice.johnson@example.com\", \"phone\": \"+1-555-123-4567\"}, \"address\": {\"street\": \"123 Maple St\", \"city\": \"Springfield\", \"state\": \"IL\", \"zip\": \"62701\", \"country\": \"USA\"}, \"personal_preferences\": {\"hobbies\": [\"reading\", \"hiking\", \"photography\"], \"favorite_food\": \"sushi\", \"music_genres\": [\"jazz\", \"classical\", \"indie\"]}, \"interesting_information\": {\"pet\": {\"type\": \"dog\", \"name\": \"Buddy\", \"breed\": \"Golden Retriever\"}, \"travel_history\": [{\"country\": \"Japan\", \"year\": 2019}, {\"country\": \"Italy\", \"year\": 2021}]}}'),\n",
      " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 2}, page_content='{\"name\": \"Michael Smith\", \"age\": 34, \"contact_details\": {\"email\": \"michael.smith@example.com\", \"phone\": \"+1-555-234-5678\"}, \"address\": {\"street\": \"456 Oak Ave\", \"city\": \"Denver\", \"state\": \"CO\", \"zip\": \"80202\", \"country\": \"USA\"}, \"personal_preferences\": {\"hobbies\": [\"cycling\", \"cooking\", \"gaming\"], \"favorite_food\": \"pizza\", \"music_genres\": [\"rock\", \"pop\", \"hip-hop\"]}, \"interesting_information\": {\"pet\": {\"type\": \"cat\", \"name\": \"Whiskers\", \"breed\": \"Siamese\"}, \"volunteering\": {\"organization\": \"Local Food Bank\", \"years_active\": 5}}}'),\n",
      " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 3}, page_content='{\"name\": \"Emily Davis\", \"age\": 22, \"contact_details\": {\"email\": \"emily.davis@example.com\", \"phone\": \"+1-555-345-6789\"}, \"address\": {\"street\": \"789 Pine Rd\", \"city\": \"Austin\", \"state\": \"TX\", \"zip\": \"73301\", \"country\": \"USA\"}, \"personal_preferences\": {\"hobbies\": [\"painting\", \"traveling\", \"yoga\"], \"favorite_food\": \"tacos\", \"music_genres\": [\"country\", \"folk\", \"dance\"]}, \"interesting_information\": {\"pet\": null, \"study\": {\"major\": \"Fine Arts\", \"university\": \"University of Texas\", \"graduation_year\": 2024}}}'),\n",
      " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 4}, page_content='{\"name\": \"David Brown\", \"age\": 45, \"contact_details\": {\"email\": \"david.brown@example.com\", \"phone\": \"+1-555-456-7890\"}, \"address\": {\"street\": \"101 Birch Blvd\", \"city\": \"Seattle\", \"state\": \"WA\", \"zip\": \"98101\", \"country\": \"USA\"}, \"personal_preferences\": {\"hobbies\": [\"golf\", \"reading\", \"fishing\"], \"favorite_food\": \"steak\", \"music_genres\": [\"blues\", \"classic rock\", \"jazz\"]}, \"interesting_information\": {\"pet\": {\"type\": \"bird\", \"name\": \"Tweety\", \"breed\": \"Canary\"}, \"career\": {\"job_title\": \"Software Engineer\", \"years_experience\": 20}}}'),\n",
      " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 5}, page_content='{\"name\": \"Sophia Wilson\", \"age\": 39, \"contact_details\": {\"email\": \"sophia.wilson@example.com\", \"phone\": \"+1-555-567-8901\"}, \"address\": {\"street\": \"202 Cedar Ct\", \"city\": \"Miami\", \"state\": \"FL\", \"zip\": \"33101\", \"country\": \"USA\"}, \"personal_preferences\": {\"hobbies\": [\"dancing\", \"gardening\", \"cooking\"], \"favorite_food\": \"paella\", \"music_genres\": [\"latin\", \"pop\", \"salsa\"]}, \"interesting_information\": {\"pet\": {\"type\": \"dog\", \"name\": \"Max\", \"breed\": \"Bulldog\"}, \"travel_history\": [{\"country\": \"Spain\", \"year\": 2018}, {\"country\": \"Brazil\", \"year\": 2020}]}}')]\n"
     ]
    }
   ],
   "source": [
    "from langchain_community.document_loaders import JSONLoader\n",
    "\n",
    "# Create JSONLoader\n",
    "loader = JSONLoader(\n",
    "    file_path=\"data/people.json\",\n",
    "    jq_schema=\".people[]\",  # Access each item in the people array\n",
    "    text_content=False,\n",
    ")\n",
    "\n",
    "# Load documents\n",
    "docs = loader.load()\n",
    "pprint(docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading Each Person as a Separate Document\n",
    "\n",
    "We can load each person object from ```people.json``` as an individual document using the ```jq_schema=\".people[]\"```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-17T13:45:45.962486Z",
     "start_time": "2025-01-17T13:45:45.948570Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 1}, page_content='{\"name\": \"Alice Johnson\", \"age\": 28, \"contact_details\": {\"email\": \"alice.johnson@example.com\", \"phone\": \"+1-555-123-4567\"}, \"address\": {\"street\": \"123 Maple St\", \"city\": \"Springfield\", \"state\": \"IL\", \"zip\": \"62701\", \"country\": \"USA\"}, \"personal_preferences\": {\"hobbies\": [\"reading\", \"hiking\", \"photography\"], \"favorite_food\": \"sushi\", \"music_genres\": [\"jazz\", \"classical\", \"indie\"]}, \"interesting_information\": {\"pet\": {\"type\": \"dog\", \"name\": \"Buddy\", \"breed\": \"Golden Retriever\"}, \"travel_history\": [{\"country\": \"Japan\", \"year\": 2019}, {\"country\": \"Italy\", \"year\": 2021}]}}'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 2}, page_content='{\"name\": \"Michael Smith\", \"age\": 34, \"contact_details\": {\"email\": \"michael.smith@example.com\", \"phone\": \"+1-555-234-5678\"}, \"address\": {\"street\": \"456 Oak Ave\", \"city\": \"Denver\", \"state\": \"CO\", \"zip\": \"80202\", \"country\": \"USA\"}, \"personal_preferences\": {\"hobbies\": [\"cycling\", \"cooking\", \"gaming\"], \"favorite_food\": \"pizza\", \"music_genres\": [\"rock\", \"pop\", \"hip-hop\"]}, \"interesting_information\": {\"pet\": {\"type\": \"cat\", \"name\": \"Whiskers\", \"breed\": \"Siamese\"}, \"volunteering\": {\"organization\": \"Local Food Bank\", \"years_active\": 5}}}'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 3}, page_content='{\"name\": \"Emily Davis\", \"age\": 22, \"contact_details\": {\"email\": \"emily.davis@example.com\", \"phone\": \"+1-555-345-6789\"}, \"address\": {\"street\": \"789 Pine Rd\", \"city\": \"Austin\", \"state\": \"TX\", \"zip\": \"73301\", \"country\": \"USA\"}, \"personal_preferences\": {\"hobbies\": [\"painting\", \"traveling\", \"yoga\"], \"favorite_food\": \"tacos\", \"music_genres\": [\"country\", \"folk\", \"dance\"]}, \"interesting_information\": {\"pet\": null, \"study\": {\"major\": \"Fine Arts\", \"university\": \"University of Texas\", \"graduation_year\": 2024}}}'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 4}, page_content='{\"name\": \"David Brown\", \"age\": 45, \"contact_details\": {\"email\": \"david.brown@example.com\", \"phone\": \"+1-555-456-7890\"}, \"address\": {\"street\": \"101 Birch Blvd\", \"city\": \"Seattle\", \"state\": \"WA\", \"zip\": \"98101\", \"country\": \"USA\"}, \"personal_preferences\": {\"hobbies\": [\"golf\", \"reading\", \"fishing\"], \"favorite_food\": \"steak\", \"music_genres\": [\"blues\", \"classic rock\", \"jazz\"]}, \"interesting_information\": {\"pet\": {\"type\": \"bird\", \"name\": \"Tweety\", \"breed\": \"Canary\"}, \"career\": {\"job_title\": \"Software Engineer\", \"years_experience\": 20}}}'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 5}, page_content='{\"name\": \"Sophia Wilson\", \"age\": 39, \"contact_details\": {\"email\": \"sophia.wilson@example.com\", \"phone\": \"+1-555-567-8901\"}, \"address\": {\"street\": \"202 Cedar Ct\", \"city\": \"Miami\", \"state\": \"FL\", \"zip\": \"33101\", \"country\": \"USA\"}, \"personal_preferences\": {\"hobbies\": [\"dancing\", \"gardening\", \"cooking\"], \"favorite_food\": \"paella\", \"music_genres\": [\"latin\", \"pop\", \"salsa\"]}, \"interesting_information\": {\"pet\": {\"type\": \"dog\", \"name\": \"Max\", \"breed\": \"Bulldog\"}, \"travel_history\": [{\"country\": \"Spain\", \"year\": 2018}, {\"country\": \"Brazil\", \"year\": 2020}]}}')]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loader = JSONLoader(\n",
    "    file_path=\"data/people.json\",\n",
    "    jq_schema=\".people[]\",\n",
    "    text_content=False,\n",
    ")\n",
    "\n",
    "data = loader.load()\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using ```content_key``` within ```jq_schema```\n",
    "\n",
    "To load documents from a JSON file using ```content_key``` within the ```jq_schema```, set ```is_content_key_jq_parsable=True```. Ensure that ```content_key``` is compatible and can be parsed using the ```jq_schema```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-17T13:46:04.815512Z",
     "start_time": "2025-01-17T13:46:04.804648Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 1}, page_content='Alice Johnson'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 2}, page_content='Michael Smith'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 3}, page_content='Emily Davis'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 4}, page_content='David Brown'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 5}, page_content='Sophia Wilson')]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loader = JSONLoader(\n",
    "    file_path=\"data/people.json\",\n",
    "    jq_schema=\".people[]\",\n",
    "    content_key=\"name\",\n",
    "    text_content=False\n",
    ")\n",
    "\n",
    "data = loader.load()\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extracting Metadata from ```people.json```\n",
    "\n",
    "Let's define a ```metadata_func``` to extract relevant information like name, age, and city from each person object.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-17T13:47:24.305343Z",
     "start_time": "2025-01-17T13:47:24.292618Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 1, 'name': 'Alice Johnson', 'age': 28, 'city': 'Springfield'}, page_content='Alice Johnson'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 2, 'name': 'Michael Smith', 'age': 34, 'city': 'Denver'}, page_content='Michael Smith'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 3, 'name': 'Emily Davis', 'age': 22, 'city': 'Austin'}, page_content='Emily Davis'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 4, 'name': 'David Brown', 'age': 45, 'city': 'Seattle'}, page_content='David Brown'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 5, 'name': 'Sophia Wilson', 'age': 39, 'city': 'Miami'}, page_content='Sophia Wilson')]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def metadata_func(record: dict, metadata: dict) -> dict:\n",
    "    metadata[\"name\"] = record.get(\"name\")\n",
    "    metadata[\"age\"] = record.get(\"age\")\n",
    "    metadata[\"city\"] = record.get(\"address\", {}).get(\"city\")\n",
    "    return metadata\n",
    "\n",
    "loader = JSONLoader(\n",
    "    file_path=\"data/people.json\",\n",
    "    jq_schema=\".people[]\",\n",
    "    content_key=\"name\",\n",
    "    metadata_func=metadata_func,\n",
    "    text_content=False\n",
    ")\n",
    "\n",
    "data = loader.load()\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Understanding JSON Query Syntax\n",
    "\n",
    "Let's explore the basic syntax of jq-style queries used in ```JSONLoader```:\n",
    "\n",
    "Basic Selectors\n",
    "   - **```.```** : Current object\n",
    "   - **```.key```** : Access specific key in object\n",
    "   - **```.[]```** : Iterate over array elements\n",
    "\n",
    "Pipe Operator\n",
    "   - **```|```** : Pass result of left expression as input to right expression\n",
    "   \n",
    "Object Construction\n",
    "   - **```{key: value}```** : Create new object\n",
    "\n",
    "Example JSON:\n",
    "```json\n",
    "{\n",
    "  \"people\": [\n",
    "    {\"name\": \"Alice\", \"age\": 30, \"contactDetails\": {\"email\": \"alice@example.com\", \"phone\": \"123-456-7890\"}},\n",
    "    {\"name\": \"Bob\", \"age\": 25, \"contactDetails\": {\"email\": \"bob@example.com\", \"phone\": \"098-765-4321\"}}\n",
    "  ]\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Common Query Patterns**:\n",
    "- ```.people[]``` : Access each array element\n",
    "- ```.people[].name``` : Get all names\n",
    "- ```.people[] | {name: .name}``` : Create new object with name\n",
    "- ```.people[] | {name, email: .contact.email}``` : Extract nested data\n",
    "\n",
    "[Note] \n",
    "- Always use ```text_content=False``` when working with complex JSON data\n",
    "- This ensures proper handling of non-string values (objects, arrays, numbers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Advanced Queries\n",
    "\n",
    "Here are examples of extracting specific information using different jq schemas:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-17T13:48:24.422895Z",
     "start_time": "2025-01-17T13:48:24.405950Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 1}, page_content='{\"name\": \"Alice Johnson\", \"contact\": null}'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 2}, page_content='{\"name\": \"Michael Smith\", \"contact\": null}'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 3}, page_content='{\"name\": \"Emily Davis\", \"contact\": null}'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 4}, page_content='{\"name\": \"David Brown\", \"contact\": null}'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 5}, page_content='{\"name\": \"Sophia Wilson\", \"contact\": null}')]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Extract only contact details\n",
    "contact_loader = JSONLoader(\n",
    "    file_path=\"data/people.json\",\n",
    "    jq_schema=\".people[] | {name: .name, contact: .contactDetails}\",\n",
    "    text_content=False\n",
    ")\n",
    "\n",
    "docs = contact_loader.load()\n",
    "docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-17T13:48:46.562120Z",
     "start_time": "2025-01-17T13:48:46.539169Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 1}, page_content='{\"name\": \"Alice Johnson\", \"hobbies\": null}'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 2}, page_content='{\"name\": \"Michael Smith\", \"hobbies\": null}'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 3}, page_content='{\"name\": \"Emily Davis\", \"hobbies\": null}'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 4}, page_content='{\"name\": \"David Brown\", \"hobbies\": null}'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 5}, page_content='{\"name\": \"Sophia Wilson\", \"hobbies\": null}')]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Extract nested data\n",
    "hobbies_loader = JSONLoader(\n",
    "    file_path=\"data/people.json\",\n",
    "    jq_schema=\".people[] | {name: .name, hobbies: .personalPreferences.hobbies}\",\n",
    "    text_content=False\n",
    ")\n",
    "\n",
    "docs = hobbies_loader.load()\n",
    "docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-17T13:48:50.327954Z",
     "start_time": "2025-01-17T13:48:50.296720Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 1}, page_content='{\"name\": \"Alice Johnson\", \"facts\": null}'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 2}, page_content='{\"name\": \"Michael Smith\", \"facts\": null}'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 3}, page_content='{\"name\": \"Emily Davis\", \"facts\": null}'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 4}, page_content='{\"name\": \"David Brown\", \"facts\": null}'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 5}, page_content='{\"name\": \"Sophia Wilson\", \"facts\": null}')]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get all interesting facts\n",
    "facts_loader = JSONLoader(\n",
    "    file_path=\"data/people.json\",\n",
    "    jq_schema=\".people[] | {name: .name, facts: .interestingFacts}\",\n",
    "    text_content=False\n",
    ")\n",
    "\n",
    "docs = facts_loader.load()\n",
    "docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-17T13:48:53.086930Z",
     "start_time": "2025-01-17T13:48:53.057731Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 1}, page_content='{\"name\": \"Alice Johnson\", \"contact\": null}'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 2}, page_content='{\"name\": \"Michael Smith\", \"contact\": null}'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 3}, page_content='{\"name\": \"Emily Davis\", \"contact\": null}'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 4}, page_content='{\"name\": \"David Brown\", \"contact\": null}'),\n",
       " Document(metadata={'source': '/Users/leejungbin/Downloads/LangChain-OpenTutorial/06-DocumentLoader/data/people.json', 'seq_num': 5}, page_content='{\"name\": \"Sophia Wilson\", \"contact\": null}')]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Extract email and phone together\n",
    "contact_info = JSONLoader(\n",
    "    file_path=\"data/people.json\",\n",
    "    jq_schema='.people[] | {name: .name, email: .contactDetails.email, phone: .contactDetails.phone}',\n",
    "    text_content=False\n",
    ")\n",
    "\n",
    "docs = contact_loader.load()\n",
    "docs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These examples demonstrate the flexibility of jq queries in fetching data in various ways."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "langchain-opentutorial-LGorndcz-py3.11",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
